Papers with Subword tokenization
Analyzing Cognitive Plausibility of Subword Tokenization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of subword tokenization focus on engineering criteria such as compression rate . a recent study evaluated subwords for their cognitive plausibility in languages with limited vocabulary size . |
| Approach: | They propose a new evaluation paradigm that focuses on the cognitive plausibility of subword tokenization. |
| Outcome: | The proposed tokenization algorithm yields less cognitively plausible tokenization behavior and worse coverage of derivational morphemes than previous evaluations. |
Tokenization and the Noiseless Channel (2023.acl-long)
Copied to clipboard
| Challenge: | Subword tokenization is a key part of most NLP pipelines, but little is known about why some combinations lead to improved downstream model performance. |
| Approach: | They propose that good tokenizers lead to efficient channel usage . they propose that an optimal encoding assigns extremely long codes to low-frequency subwords . |
| Outcome: | The proposed tokenizers have a very strong correlation with BLEU in machine translation . the proposed function can be used to improve model performance in the downstream task . |
Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Subword tokenization is a common method for vocabulary building in NMT systems . but, it has its disadvantages, such as a vocabulary cannot be modified once it is learned . |
| Approach: | They propose a method that learns contextualized information of varying scales . they propose byte-based tokenization to solve these problems with few embedding parameters . |
| Outcome: | Experiments show that the proposed method outperforms subword-based methods in multilingual and out-of-domain scenarios. |
How Important Is Tokenization in French Medical Masked Language Models? (2024.lrec-main)
Copied to clipboard
| Challenge: | Word tokenization into subword units has become the prevailing standard in the field of natural language processing (NLP) over recent years . the precise factors contributing to its success remain unclear . |
| Approach: | They propose a tokenization strategy that integrates morpheme-enriched word segmentation into existing tokenization methods. |
| Outcome: | The proposed tokenization strategy outperforms character and word tokenization but the precise factors contributing to its success remain unclear. |